From: "Blair Taylor" <btaylor(3)memphistomorrow.org> 

To: "Angela Madden (acrewsi)" <acrews1(2)memphis.edu> 

"William Gibbons \(wgibbons\)" <wgibbons(2)memphis.edu> 

"Brona Pinnolis" <bpinnolis@memphiscrime.org> 

"Weirich, Amy" <Amy.Weirich@scdag.com> 
wgiibbons(2)memphis.edu 
"Mike Gates" <Mike.Gates(2)shelbycountytn.gov> 
kevin.woods(a)workforceinvestmentnetwork.com 
Mike.Shearin(a)memphistn.gov 
Julie.Nations(2)memphistn.gov 
Date: 6/13/2017 11:55:35 AM 

Subject: RE: Very convincing rationale for using randomized controlled trials 


Great info! Thanks for sharing! 

Blair Taylor 
President 

Memphis Tomorrow 

22 North Front Street, Suite 670 

Memphis,TN 38103 

Office: 901.322.8080 

btaylor© memphistomorrow.org 

From: Angela Madden (acrewsi) [mailto:acrewsl@memphis.edu] 

Sent: Tuesday, June 13, 2017 10:47 AM 

To: William Gibbons (wgibbons) <wgibbons@memphis.edu>; Brona Pinnolis <bpinnolis@memphiscrime.org>; Weirich, 
Amy <Amy.Weirich@scdag.com>; wgiibbons@memphis.edu; M ike Gates <fV| ike.Gates@shelbycountytn.gov>; 
kevin.woods@workforceinvestmentnetwork.com; Blair Taylor <btaylor@memphistomorrow.org>; 

M ike.Shearin@memphistn.gov; Julie.Nations@memphistn.gov 
Subject: Very convincing rationale for using randomized controlled trials 

Morning all! 

Very interesting class this morning from a Harvard economist illustrating why randomized controlled trials are the 
best way to determine program impact. I think that some of this material could prove helpful to various agencies 
in our area who are implementing programs and whose funders want evidence of program impact (or 
government agencies accountable to taxpayers). Perhaps a symposium topic in the future? 

This table compares results using 6 different methodological techniques to determine the impact of a "get out the 
vote" phone call campaign on voting behavior (voting in 1998 compared to voting in 2002). 

The "Pre-Post" method below simply compares the % of voters in 2002 who were called and urged to "get out 
and vote" to the % of those same people who voted in 1998. This method shows a 17.9 percentage point 
increase. As the method gets more rigorous and controlled, that percentage point difference goes down although 
they are all "statistically significant" until you get to the "Randomized Evaluation" (the most rigorous method). 

The difference between voting in 2002 is only .04 percentage points higher than 1998 and it is NOT statistically 
significant. 



Table 1: Comparing all six methods 


Method 

Estimated 

impact 

Pre-Post 

17.9 

PP* 

Simple Differenee 

10.8 

PP* 

Difference-in-Differences 

1.9 

PP* 

Multivariate Regression with 

4 fi 


Panel Data 


PP 

Matehing (All Covariates) 

2.8 

PP* 

Randomized Evaluation 

0.4 

PP 


NOTES: pp means ope re entage pointsoand * indicates statistic ally significant at the 5% level 

A Randomized evaluation estimate is adjusted to reflect that only 25,000 of 60,000 in the treatment were treated (because they answered the phone call; 35,000 in the 
'treatmentgroup"did notanswerthe phone) 

just wanted to share! 1 

Thanks, 

AM 


Angela D. Madden, Ph.D. 
Research Associate Professor 
Public Safety Institute 



The University of Memphis 
321 McCord Hall 
Memphis, TN 38152 

901.678.5923 | acrews1@memphis.edu 












